Add an sshfs backend to eval-under - #13

The backend was on-demand only, so nothing watched it. Add it as a matrix
row: `sshfs (loopback) / git-annex test` and `sshfs (loopback) / git
testsuite`, taking the matrix from 20 cells to 22.

Not four cells. stress-ng and pjdfstest are needs-root, and a FUSE mount
belongs to whoever mounted it -- there is no --no-root-squash equivalent
the way there is for NFS -- so those two could only ever report on
privilege rather than on the filesystem. run-under.sh already refuses
that pair with exit 2; the matrix now expresses the same rule as data.

Rather than a hand-kept exclusion list, the backend row carries
`no-root: true` and `cell_enabled()` in matrix.sh derives the gap from it
plus the target's existing `needs-root`. So the reason lives in one place,
a future unprivileged backend gets the behaviour for free, and the grid
cannot silently grow a cell that measures privilege. Every consumer asks
`cell_enabled` instead of assuming backends x targets:

- matrix-json.sh omits the pair, so the workflow never schedules it;
- gen-readme-matrix.sh prints `n/a` instead of a badge (README regenerated);
- update-status.py keeps it out of status.json, so it publishes no
  permanently-unknown badge;
- render-report.py renders an explicit gap rather than an "unknown" badge,
  which would read as "not measured yet".

`sshfs-git-annex` is expected red and is annotated as such on the report
page and in GOTCHAS.md: invisible hardlinks (link() succeeds, st_ino
differs, nlink=1) break `git annex add`'s post-link verification.
`sshfs-git` is the control -- same mount, a suite that never hardlinks
into an object store. Its result is not predicted here; the first run
measures it, and GOTCHAS.md gets the answer either way.

Verified: shellcheck clean and 28/28 bats; matrix-json.sh emits 22 cells
with exactly the two sshfs ones; update-status.py's cell set agrees and
omits sshfs-stress-ng / sshfs-pjdfstest; and render-report.py, run over a
fabricated 22-cell status.json, writes 22 cell badges plus overall, shows
two n/a gaps in the sshfs row, and carries the expected-red note on
sshfs-git-annex.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E1fDiGVWKywpbhVps5hzQK

The `sshfs / git testsuite` cell came back red, and the previous commit
had called it "the control: a suite that never hardlinks into an object
store". That was wrong, and measuring it turned up a bug in our own
reporting.

Root cause of the cell: `git clone <local path>` hardlinks each object
and then sanity-checks the result, comparing st_mode/st_ino/st_dev/
st_size/st_uid/st_gid against the source (builtin/clone.c). sshfs
synthesises st_ino per path, so the check fails and the clone dies with
"hardlink different from source". Plain `git clone` of a local path does
not work on sshfs at all -- a broader statement than the git-annex one,
and the same mechanism. It accounts for t1507-rev-parse-upstream (20),
t1013-read-tree-submodule (58) and t0035-safe-bare-repository (2), each
of which clones or adds a submodule in setup. A second, independent cause
is the absence of unix sockets: t0301-credential-cache (37, "unable to
bind ... Operation not permitted") and t0052-simple-ipc (9/9).

`-o disable_hardlink` fixes both this and the git-annex failure, which is
the counter-intuitive part worth telling a reporter: sshfs then fails
link() with EPERM instead of pretending, and every caller here has a copy
fallback that only an honest failure reaches. Measured, with an ext4
control:

                                     default  disable_hardlink  ext4
  git annex add (locked)                 ok         ok            ok
  git annex add, annex.addunlocked=true  FAIL       ok            ok
  git clone <local path>                 FAIL       ok            ok

The reporting bug: `not ok N ... # TODO known breakage` is git's
test_expect_failure -- a TAP TODO directive that prove counts as an
expected result, not a failure. dump-failure-logs.sh counted those, so
this cell reported 351 failed assertions where prove saw 166, and the two
worst-looking scripts (t1517-outside-repo at 104, t0450-txt-doc-vs-help
at 54) are ones prove reports as *ok*. That sends triage after failures
that do not exist; it inflated every git cell, vfat included, not just
this one. Both the count and the detail listing now exclude the
directive, matching prove.

GOTCHAS.md gets the mechanisms, the workaround table, a single-script
reproduction recipe, and -- deliberately -- the list of small failures
that are *not* explained (t0003-attributes 6, t1091 8, t0610 3, t0021 3,
ten scripts with one each), so nobody assumes they share a cause.

Method: built the pinned git v2.55.0 locally and ran the whole t0*/t1*
range under the sshfs backend (174 scripts, 10366 assertions) plus an
ext4 control, which passes the five scripts under suspicion. Reproduced
the clone failure minimally, read git's check in clone.c, and confirmed
link() returns EPERM under disable_hardlink. One hypothesis was tested
and rejected rather than left implied: synthesised inode numbers do not
collide at 4000 entries, so that is not behind the whole-suite failures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E1fDiGVWKywpbhVps5hzQK

The rebase dropped my copies of this rule from update-status.py and
render-report.py: master now enumerates cells once, in matrix_cells() in
bin/ci/evals.py, which is a better home for it than the two duplicates
the pre-rebase branch had.

cell_enabled() states the rule -- a `no-root` backend has no cell for a
`needs-root` target, because such a cell could only report on privilege
rather than on the filesystem -- and matrix_cells() skips those pairs, so
status.json, the badges and the report page agree with the workflow
instead of publishing a permanently-unknown badge. render-report.py draws
an explicit `n/a` gap for them; an "unknown" badge would read as "not
measured yet". The shell side already has the same rule in matrix.sh, and
run-under.sh refuses the pair outright.

Verified: both sides report 22 cells with exactly sshfs-git-annex and
sshfs-git, and neither reports sshfs-stress-ng or sshfs-pjdfstest; the
README grid shows two n/a entries; run-checks.sh passes in full.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E1fDiGVWKywpbhVps5hzQK

The rebase put this branch on master's known-issues machinery, which
judges a cell per test rather than by the suite's exit code. The two new
sshfs cells had no entries, so they would have reported `failing-new`
even though their failures are the documented findings this backend
exists to surface. The prose that used to say so by hand in GOTCHAS.md
belongs in evals/known-issues.yaml now, where the verdict, the status
page and GOTCHAS all read it from one place.

Measured, not asserted: ran the cell here (`run-under.sh sshfs n/a git`
then `check-cell.sh`), which reported 185 failures across 30 scripts, and
attributed the scripts by the message in their own logs rather than by
guessing:

- `sshfs-git-local-clone-hardlink` (114 failures, 16 scripts,
  fs-divergence) -- `git clone <local path>` hardlinks each object then
  compares st_ino/st_dev/... against the source (builtin/clone.c), and
  sshfs synthesises st_ino per path, so the clone dies with `fatal:
  hardlink different from source`. Every script here clones or adds a
  submodule in setup. Plain `git clone` of a local path does not work on
  sshfs at all; --no-hardlinks, file:// and `-o disable_hardlink` do.
- `sshfs-git-no-unix-sockets` (46, fs-limitation) -- `bind()` on the mount
  fails with EPERM, so the credential-cache daemon never starts and
  simple-ipc finds no server. Mirrors the existing vfat entry;
  `unix-socket=no` in fs-capabilities.sh predicts it.
- `sshfs-git-untriaged` (25, needs-triage) -- kept deliberately separate:
  neither signature appears in these scripts' logs, so folding them into
  the mechanisms above would be a guess. Candidates noted (1-second mtime
  granularity, no xattr/fifo).

With these, the cell judges `failing-known` -> conclusion=success: 9734
pass, 185 fail all known, 0 new. GOTCHAS.md's generated section is
regenerated from them, and run-checks.sh passes in full.

Still outstanding: `sshfs (loopback) / git-annex test` has no entry yet,
so it will report `failing-new` on its first run here. Its test ids are
best taken from CI's own verdict -- CI runs a git-annex daily build, this
container has 10.20240129, and ids from the wrong build would mask real
failures rather than document them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E1fDiGVWKywpbhVps5hzQK

My previous commit reseeded these entries from CI run 36476334300 and
predicted the cell would go green. It did not: run 36485450259, on
near-identical code, reported 18 residual failures with **no overlap**
with the 12 that run reported. Every id I had just added passed, and ids
I had just removed as "now passing" -- t0003#41, t0017#4, t1430#26,
t1700#9 -> #10-15 -- were back. The residue is not a fixed set of
divergences, so no `script#N` list can cover it, and the explanation I
committed last time (root renumbering the scripts) was wrong: locally
t0003 yields 24,25,29,32,33,34, exactly CI's ids.

What is actually true, measured:

- The two mechanism entries are deterministic. `local-clone-hardlink`
  reproduced 110/110 and `no-unix-sockets` 46/46 on both CI runs, and
  t0003's six hardlink assertions failed in all six local runs.
- The residue is flaky at a low per-test rate. Six local runs of
  `t0003 t0017 t0040 t1700`: t0017#4 failed in one, t0040#37 and
  t1700#10-15 and t0003#41/#48 in none, though CI has flagged each.
- It needs concurrent load. `prove --jobs 1` is no cleaner than
  `--jobs 4`, and 14 runs of t0017 on its own were all clean.
- Some of it cannot be filesystem semantics at all: t0040#37
  "OPT_CALLBACK() and OPT_BIT() work" and t0017#4 "test-tool env-helper
  --type=ulong" parse arguments and environment variables, and t0450
  compares documentation against `-h` output. What they share is
  capturing output into `>out`/`2>err` in the trash directory on the
  mount and grepping it, which points at the mount losing or delaying
  writes under load.

So this commit stops pretending the list gates anything. The entry now
carries the union observed across both CI runs, labelled as a sample,
plus the measurements above and the untried candidate that would
actually resolve it: mounting with `-o attr_timeout=0 -o entry_timeout=0`
(possibly with `-o max_conns=N` or `-o sync_read`) and re-measuring,
which if it works makes the cell deterministic and properly xfail-able.

This does NOT make the cell green -- it will still report `failing-new`
whenever a run draws a flake outside the sample, and that is now the
honest state rather than a hidden one. run-checks.sh passes and
GOTCHAS.md is regenerated.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E1fDiGVWKywpbhVps5hzQK

Two things this branch owed.

`dump-failure-logs.sh` had no `sshfs)` arm, so every failing sshfs cell
printed `unknown backend: sshfs` and no diagnostics at all -- visible in
the git-annex cell's log, which is the one place they would have helped.
The new arm prints any surviving `/tmp/eval-under-sshfs-*.scratch/sshd.log`
(the backend removes its scratch dir on teardown, so the log is only
there when teardown was skipped -- a crash, or --keep), any leftover
`mount -t fuse.sshfs`, which is itself the finding when a suite timed
out, and fuse-tagged dmesg.

And the `cache=no` candidate the last commit left untried is now
measured: five runs of `t0*.sh` at `--jobs 4`, counting failures beyond
the 64 assertions this subset's two mechanisms own. Baseline drew 4, 8
and 8; `cache=no` drew 1 and 2, at about 7% more wall clock. So sshfs's
caching is implicated, and `cache=no` is worth having as a mitigation --
but it does not make the cell green, because known_issues.py sets
`failing-new` on the first uncovered failure and has no notion of a flake
budget. One flake per run still fails the cell most runs.

That turns the remaining question into a decision rather than a patch,
and the note in evals/known-issues.yaml now says so: cover the whole
cell with `tests: ["*"]` the way `vfat-pjdfstest` does (green, but it
masks the 110 and 46 findings along with any future regression), make
the cell non-gating, or give the verdict machinery a flake budget --
which belongs in the machinery, not in this backend's PR. Not choosing
one here on my own.

shellcheck clean over 26 scripts; run-checks.sh passes; GOTCHAS.md
regenerated.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E1fDiGVWKywpbhVps5hzQK